Recording medium, recording method, reproduction apparatus and method, and computer-readable program

ABSTRACT

An AV Clip generated by multiplexing video and graphics streams is recorded on a BD-ROM. The graphics stream is a sequence of PES packets. The PES packets include ODSs (Object Definition Segments) carrying graphics data and PCSs (Presentation Control Segments) carrying control information. A PCS indicates that graphics data in a preceding ODS in the arrangement is to be displayed at a predetermined time in a state of being overlaid on the video stream. The PCS and the preceding ODS belong to different DSs, and the graphics data in the preceding ODS may be non-referenced graphics data that is not referenced by a PCS which belongs to the same DS as the preceding ODS.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a divisional application of U.S. Ser. No. 10/561,087, filed onDec. 16, 2005.

TECHNICAL FIELD

The present invention relates to a recording medium on which a digitalstream generated by multiplexing a video stream and a graphics stream isrecorded, and a reproduction apparatus for reproducing the digitalstream. In particular, the present invention relates to techniqueswhereby the reproduction apparatus separately decodes the video streamand the graphics stream and overlays resulting video and graphics toproduce a reproduction image.

BACKGROUND ART

Video/graphics overlay techniques mentioned above enable to selectgraphics and display the selected graphics in accordance with a languagesetting or display setting of a reproduction apparatus. Also, thevideo/graphics overlay techniques enable to select whether or not todisplay graphics according to need. Because of their capability ofoffering such selections, the video/graphics overlay techniques areemployed in player models of DVD-Video and ETSI EN 300 743. In thesestandards, the graphics stream is an arrangement of PES packets. The PESpackets can be classified into two types: a PES packet carrying controlinformation and a PES packet carrying graphics data which representsgraphics. The control information is arranged before the graphics data,and one pair of control information and graphics data produces onegraphics display. In more detail, the reproduction apparatus reads thecontrol information and the graphics data in sequence, decodes thecontrol information, and also decodes the graphics data to generateuncompressed graphics. The reproduction apparatus displays theuncompressed graphics with a desired presentation timing, according to adecoding result of the control information.

According to this technique, decoding of graphics data needs to beperformed for each graphics display. This being so, a time interval fromone graphics display to the next graphics display depends on a timeperiod required for a decoding process of graphics data. If graphics hasa higher resolution and a decoding process of graphics data takes alonger time, a time interval with which graphics is updated lengthens.

In the case of subtitling in a movie film or the like where graphics isupdated with a time interval of 2 to 3 seconds, proper graphics displayscan still be produced even if graphics has a higher resolution anddecoding of graphics data takes a longer time. In the case wheregraphics is used specially as a means of amusement, however, the need toshorten the graphics update interval arises.

For example, the use of graphics for amusement purposes includes suchapplications where graphics representing a dialog of a character ismoved to attract the viewer's attention or varied in color to surprisethe viewer. Producers of video content such as variety shows areespecially eager to adopt such graphics displays.

To realize smooth movements of graphics, however, an operation ofdecoding graphics data and presenting graphics needs to be performedwithin a short time of one frame period (1/29.97 seconds in NTSC).Repeating this operation of decoding graphics data and presentinggraphics within each frame period requires a tremendous load, and ishard to be realized with hardware specifications of reproductionapparatuses that are expected to be provided at low cost.

The aforementioned graphics movements and variations in color can beachieved by overlaying graphics on each frame of video beforehand sothat graphics forms a part of a picture pattern of each frame. Thismethod of overlaying graphics on each frame beforehand, however, lacksflexibility of being able to selectively display graphics in accordancewith a language setting or display setting of a reproduction apparatus,and therefore does not have much future potential.

DISCLOSURE OF INVENTION

The present invention aims to provide a recording medium which enablesgraphics to be moved smoothly like moving pictures.

The stated aim can be achieved by a recording medium used for storingdata, including a digital stream generated by multiplexing a videostream and a graphics stream, wherein: the graphics stream is a sequenceof a plurality of packets which include a packet containing controlinformation; and the control information indicates that graphics datacontained in a preceding packet in the sequence is to be displayed at apredetermined time in a state of being overlaid on the video stream.

Control information indicates that a display is to be produced usinggraphics data that precedes the control information. Accordingly, adisplay position of graphics can be changed by feeding only controlinformation showing new coordinates of the graphics into a reproductionapparatus. Equally, a color of graphics can be changed by feeding onlyappropriate control information into the reproduction apparatus.

Since a graphics display can be updated using only control information,graphics is easily brought into synchronization with video.

Usually, a picture pattern of graphics itself need not be varied inorder to change a display position of the graphics. This is because thehuman eye cannot perceive a moving object so distinctly. This being so,the technique provided by the present invention is suitable fordisplaying the same graphics while changing its position at high speed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows an example application of a recording medium to whichembodiments of the present invention relate.

FIG. 2 shows a structure of a BD-ROM shown in FIG. 1.

FIG. 3 shows a structure of an AV Clip.

FIG. 4A shows a structure of a Presentation graphics stream.

FIG. 4B shows PES packets which contain functional Segments.

FIG. 5 shows a logical structure made up of various types of functionalSegments.

FIG. 6 shows a relationship between subtitle display positions andEpochs.

FIG. 7A shows a data structure of an ODS.

FIG. 7B shows a data structure of a PDS.

FIG. 8A shows a data structure of a WDS.

FIG. 8B shows a data structure of a PCS.

FIG. 9 shows an example description of DSs for displaying subtitles.

FIG. 10 shows an example description of a PCS and a WDS in DS1.

FIG. 11 shows an example description of a PCS in DS2.

FIG. 12 shows an example description of a PCS in DS3.

FIG. 13 shows a memory space in an Object Buffer when performinggraphics updates such as those shown in FIGS. 10 to 12.

FIG. 14 shows an example algorithm of calculating a DECODEDURATION.

FIG. 15 is a flowchart of the algorithm shown in FIG. 14.

FIGS. 16A and 16B are flowcharts of the algorithm shown in FIG. 14.

FIG. 17A shows a situation where one graphics Object exists in oneWindow.

FIGS. 17B and 17C are timing charts showing parameters used in thealgorithm shown in FIG. 14.

FIG. 18A shows a situation where two graphics Objects exist in oneWindow.

FIGS. 18B and 18C are timing charts showing parameters used in thealgorithm shown in FIG. 14.

FIG. 19A shows a situation where two graphics Objects exist respectivelyin two Windows.

FIG. 19B is a timing chart when decode period (2) is longer than a sumof clear period (1) and write period (31).

FIG. 19C is a timing chart when the sum of clear period (1) and writeperiod (31) is longer than decode period (2).

FIG. 20 shows forward reference PCSs.

FIG. 21A shows an Epoch that produces a display effect of movinggraphics on a screen.

FIG. 21B shows contents and PTSs of PCSs in DS1 to DS8.

FIG. 22A shows an ODS in DS0.

FIG. 22B shows positions of coordinates (x1,y1), (x2,y2), (x3,y3), . . ., (x8,y8) in a coordinate system defined by a Window.

FIG. 23 shows a settings of a time stamp of a functional Segments ineach DS.

FIG. 24 shows a specific example of successive graphics updates.

FIG. 25 shows a series of DSs for realizing a Palette-only DisplayUpdate.

FIG. 26A shows contents of PDSs in DS0.

FIG. 26B shows contents of PCSs in DS0 to DS3.

FIG. 27 shows a display effect achieved by feeding the four DSs.

FIG. 28 shows an internal construction of a reproduction apparatus towhich the embodiments of the present invention relate.

FIG. 29 shows transfer rates Rx, Rc, and Rd and sizes of a GraphicsPlane, a Coded Data Buffer, and an Object Buffer shown in FIG. 28.

FIG. 30 is a timing chart of pipeline processing in the reproductionapparatus.

FIG. 31 is a timing chart of pipeline processing when decoding of ODSsends before clearing of the Graphics Plane.

FIG. 32 is a timing chart showing changes in occupancy of theComposition Buffer, the Object Buffer, the Coded Data Buffer, and theGraphics Plane.

FIG. 33 is a flowchart of an operation of loading functional Segments.

FIG. 34 shows a case when a skip operation is performed.

FIG. 35 shows a situation where DS10 is loaded into the Coded DataBuffer when the skip operation is performed as shown in FIG. 34.

FIG. 36 shows a case when normal reproduction is performed.

FIG. 37 shows a situation where DS1 and DS20 are loaded into the CodedData Buffer when the normal reproduction is performed as shown in FIG.36.

FIG. 38 is a flowchart of an operation of a Graphics Controller shown inFIG. 28.

FIG. 39 is a flowchart of the operation of the Graphics Controller.

FIG. 40 is a flowchart of the operation of the Graphics Controller.

FIG. 41 shows manufacturing steps of the BD-ROM.

BEST MODE FOR CARRYING OUT THE INVENTION First Embodiment

The following is a description on a recording medium to which a firstembodiment of the present invention relates. First, use of the recordingmedium is explained below. FIG. 1 shows an example application of therecording medium. In the drawing, the recording medium is a BD-ROM 100.The BD-ROM 100 is used for providing a movie film in a home theatersystem that includes a reproduction apparatus 200, a television 300, anda remote control 400.

Production of the recording medium is explained next. The recordingmedium can be realized by making improvements to an application layer ofa BD-ROM. FIG. 2 shows an example structure of the BD-ROM 100.

In the drawing, the fourth level shows the BD-ROM 100, and the thirdlevel shows a track on the BD-ROM 100. The track is shown as beingstretched out into a straight line, though in actuality the trackspirals outwards from the center of the BD-ROM 100. The track includes alead-in area, a volume area, and a lead-out area. The volume area has alayer model of a physical layer, a file system layer, and an applicationlayer. The first level shows a format of the application layer(application format) of the BD-ROM 100 in a directory structure. Asillustrated, the BD-ROM 100 has a BDMV directory below a ROOT directory.The BDMV directory contains a file (XXX.M2TS) storing an AV Clip, a file(XXX.CLPI) storing management information of the AV Clip, and a file(YYY.MPLS) defining a logical playback path (playlist) for the AV Clip.The BD-ROM 100 can be realized by generating such an application format.If there are more than one file for each of the above file types, threedirectories named STREAM, CLIPINF, and PLAYLIST may be provided belowthe BDMV directory, to store files of the same type as XXX.M2TS, filesof the same type as XXX.CLPI, and files of the same type as YYY.MPLSrespectively.

The AV Clip (XXX.M2TS) in this application format is explained below.

The AV Clip (XXX.M2TS) is a digital stream of the MPEG-TS (TransportStream) format, and is obtained by multiplexing a video stream, at leastone audio stream, and a Presentation graphics stream. The video streamrepresents a moving picture of the film, the audio stream representsaudio of the film, and the Presentation graphics stream representssubtitles of the film. FIG. 3 shows a structure of the AV Clip(XXX.M2TS).

In the drawing, the middle level shows the AV Clip. This AV Clip can becreated as follows. The video stream made up of a plurality of videoframes (pictures pj1, pj2, pj3, . . . ) and the audio stream made up ofa plurality of audio frames on the upper first level are each convertedto PES packets on the upper second level, and further converted to TSpackets on the upper third level. Likewise, the Presentation graphicsstream on the lower first level is converted to PES packets on the lowersecond level, and further converted to TS packets on the lower thirdlevel. These TS packets of the video, audio, and Presentation graphicsstreams are multiplexed to form the AV Clip.

FIG. 3 shows an example where only one Presentation graphics stream ismultiplexed in the AV Clip. If the BD-ROM 100 supports multiplelanguages, however, a Presentation graphics stream for each of thelanguages is multiplexed in the AV Clip. The AV Clip generated in theabove manner is divided into a plurality of extents in the same way ascomputer files, and stored on the BD-ROM 100.

The following explains the Presentation graphics stream. FIG. 4A shows astructure of the Presentation graphics stream. In the drawing, the firstlevel shows the TS packet string which constitutes the AV Clip. Thesecond level shows the PES packet string which constitutes thePresentation graphics stream. This PES packet string is formed byconnecting payloads of TS packets having a predetermined PID from the TSpacket string on the first level.

The third level shows the structure of the Presentation graphics stream.The Presentation graphics stream is made up of functional Segments thatinclude a PCS (Presentation Composition Segment), a WDS (WindowDefinition Segment), a PDS (Palette Definition Segment), an ODS (ObjectDefinition Segment), and an END (End of Display Set Segment). Of thesefunctional Segments, the PCS is a screen Composition Segment, whereasthe WDS, the PDS, and the ODS are Definition Segments. One functionalSegment corresponds to either one PES packet or a plurality of PESpackets. Which is to say, one functional Segment is converted to one PESpacket and recorded on the BD-ROM 100, or split into fragments which areconverted to PES packets and recorded on the BD-ROM 100.

FIG. 4B shows PES packets containing functional Segments As illustrated,each PES packet is made up of a packet header and a payload. The payloadcarries a functional Segment, and the packet header carries a DTS and aPTS associated with the functional Segment. Hereafter, a DTS and a PTSin a packet header of a PES packet which contains a functional Segmentare regarded as a DTS and a PTS of that functional Segment.

These various types of functional Segments form a logical structure suchas the one shown in FIG. 5. In the drawing, the third level shows thefunctional Segments, the second level shows DSs (Display Sets), and thefirst level shows Epochs.

ADS on the second level is a group of functional Segments, in thePresentation graphics stream, which are used for creating one screen ofgraphics. Dashed lines hk2 show which DS the functional Segments on thethird level belong to. As can be seen from the drawing, the series offunctional Segments PCS-WDS-PDS-ODS-END composes one DS. Thereproduction apparatus 200 reads these functional Segments which composethe DS from the BD-ROM 100, to produce one screen of graphics.

An Epoch on the first level refers to one time unit of continuous memorymanagement on a reproduction time axis of the AV Clip, and to a group ofdata allocated to that time unit. Memory mentioned here includes aGraphics Plane for storing one screen of graphics and an Object Bufferfor storing uncompressed graphics data. Continuous memory managementmeans that throughout the Epoch neither the Graphics Plane nor theObject Buffer is flushed and deletion and rendering of graphics areperformed only within a predetermined rectangular area of the GraphicsPlane (to flush means to clear the entire Graphics Plane or the entireObject Buffer). A size and a position of this rectangular area are fixedduring the Epoch. So long as deletion and rendering of graphics areperformed within this fixed rectangular area of the Graphics Plane,synchronization of video and graphics is guaranteed. In other words, theEpoch is a time unit, on the reproduction time axis of the AV Clip,during which synchronization of video and graphics can be guaranteed. Tochange the graphics deletion/rendering area in the Graphics Plane, it isnecessary to define a point of change on the reproduction time axis andset a new Epoch from the point onward. Synchronization of video andgraphics is not guaranteed in a boundary between the two Epochs.

In regard to subtitling, the Epoch is a time period, on the reproductiontime axis, during which subtitles appear within the fixed rectangulararea on a screen. FIG. 6 shows a relationship between subtitle displaypositions and Epochs. In the drawing, subtitle display positions arechanged depending on patterns of pictures. In more detail, threesubtitles “Actually”, “I lied to you.”, and “Sorry.” are positioned atthe bottom of the screen, whereas two subtitles “Three years havepassed” and “since then.” are positioned at the top of the screen. Thus,the subtitle display positions are changed from one margin to another onthe screen, to enhance visibility. In such a case, on the reproductiontime axis of the AV Clip, a time period during which the subtitles aredisplayed at the bottom of the screen is Epoch1, and a time periodduring which the subtitles are displayed at the top of the screen isEpoch2. These two Epochs each have an individual subtitle renderingarea. In Epoch1, the subtitle rendering area is Window1 that correspondsto the bottom margin of the screen. In Epoch2, the subtitle renderingarea is Window2 that corresponds to the top margin of the screen. Ineach of Epoch1 and Epoch2, memory management of the Object Buffer andthe Graphics Plane is continuous, so that the subtitles are displayedseamlessly in the corresponding margin of the screen. This completes theexplanation on an Epoch.

The following explains a DS.

In FIG. 5, dashed lines hk1 indicate which Epoch DSs on the second levelbelong to. As illustrated, a series of DSs that are an Epoch Start DS,an Acquisition Point DS, and a Normal Case DS constitutes one Epoch onthe first level. Here, Epoch Start, Acquisition Point, and Normal Caseare types of DSs. Though the Acquisition Point DS precedes the NormalCase DS in FIG. 5, they may be arranged in reverse order.

The Epoch Start DS provides a display effect “new display”, andindicates a start of a new Epoch. The Epoch Start DS contains allfunctional Segments necessary for the next screen composition. The EpochStart DS is provided in a position which is to be a destination of askip operation, such as a start of a chapter in a film.

The Acquisition Point DS provides a display effect “display refresh”,and is identical to the preceding Epoch Start DS. The Acquisition PointDS is not the start of the Epoch, but contains all functional Segmentsnecessary for the next screen composition. Therefore, graphics can bedisplayed reliably when reproduction is started from the AcquisitionPoint DS. Which is to say, the Acquisition Point DS enables a screencomposition to be made from a midpoint in the Epoch.

The Acquisition Point DS is provided in a position which can be adestination of a skip operation, such as a position that may bedesignated by a time search. The time search is an operation of locatinga reproduction point corresponding to a time input by a user inminutes/seconds. The time input is made in a relatively large unit suchas ten minutes or ten seconds. Accordingly, the Acquisition Point DS isprovided in a position that can be designated by a time search made inunits of ten minutes or ten seconds. By providing the Acquisition PointDS in a position that can be designated by a time search, the graphicsstream can be smoothly reproduced when a time search is conducted.

The Normal Case DS provides a display effect “display update”, andcontains only a difference from the previous screen composition. Forexample, if DSv has the same subtitle as immediately preceding DSu but adifferent screen composition from DSu, DSv is a Normal Case DS whichcontains only a PCS and an END. This makes it unnecessary to provideoverlapping ODSs in DSs, with it being possible to reduce the amount ofdata stored on the BD-ROM 100. Since the Normal Case DS contains onlythe difference, graphics cannot be displayed with the Normal Case DSalone.

The following explains the ODS, the WDS, and the PDS (DefinitionSegments).

The ODS is a functional Segment for defining a graphics Object. AV Clipsrecorded on BD-ROMs feature an image quality as high as high-definitiontelevision. This being so, graphics Objects are set at a high resolutionof 1920×1080 pixels. This high resolution allows theater screen-stylesubtitles, i.e. elegant handwriting-style subtitles, to be reproducedvividly on BD-ROMs. A graphics Object is made up of a plurality ofpieces of run-length data. Run-length data expresses a pixel stringusing a Pixel Code which shows a pixel value and a continuous length ofthe pixel value. The Pixel Code has 8 bits, and shows one of the valuesfrom 1 to 255. Through the use of this Pixel Code, the run-length datasets arbitrary 256 pixel colors out of full color (16,777,216 colors).Note that it is necessary to place a character string on a background ofa transparent color in order to display a graphics Object as a subtitle.

The ODS defines a graphics Object according to a data structure shown inFIG. 7A. As shown in the drawing, the ODS includes a segment_type fieldshowing a Segment type “ODS”, a segment_length field showing a datalength of the ODS, an object_id field identifying the graphics Object inthe Epoch, an object_version_number field showing a version of the ODSin the Epoch, a last_in_sequence_flag field, and an object_data_fragmentfield carrying a consecutive sequence of bytes corresponding to part orall of the graphics Object.

In more detail, the object_id field shows an identifier which identifiesthe graphics Object and a storage area in the Object Buffer that isoccupied by the graphics Object, when the ODS is decoded and thegraphics Object is buffered in the Object Buffer. This being so, whenone or more graphics Objects are present in the Object Buffer, eachindividual storage area in the Object Buffer is identified by anobject_id field value. Suppose one object_id is assigned to two or moreODSs. In such a case, after a graphics Object corresponding to one ODSis stored in the Object Buffer, that graphics Object is overwritten by agraphics Object corresponding to a succeeding ODS with the sameobject_id. Such an update intends to prevent occurrence of many smallfree spaces in the Object Buffer and scattering of graphics Objects inthe Object Buffer. When displaying graphics, graphics Objects in theObject Buffer are constantly transferred to the Graphics Plane. Thisbeing so, if many small free spaces exist in the Object Buffer or onegraphics Object is scattered in the Object Buffer, overhead for readinggraphics Objects causes a reduction in efficiency of transfer from theObject Buffer to the Graphics Plane. Such a reduction in transferefficiency may affect synchronous display of graphics and video. Toprevent this, an existing graphics Object in the Object Buffer is overwritten by a new graphics Object having the same object_id.

Here, the new graphics Object overwriting the existing graphics Objectneeds to be equal in size to the existing graphics Object, that is, thenew graphics Object can be neither smaller nor larger than the existinggraphics Object. At the time of authoring, therefore, an author needs tomake these graphics Objects equal in size. This size constraint thatgraphics Objects having the same object_id need be equal in width andheight applies only within an Epoch. Graphics Objects having the sameobject_id need not be equal in size if they belong to different Epochs.

The last_in_sequence_flag field and the object_data_flagment field areexplained next. Due to a constraint of payloads of PES packets,uncompressed graphics constituting one subtitle may not be able to becontained in one ODS. In such a case, the graphics is split into aplurality of fragments and one of such fragments is carried in theobject_data_fragment field. When storing one graphics Object across aplurality of ODSs, every fragment except the last fragment is of thesame size. That is, the last fragment is less than or equal to the sizeof the preceding fragments. The ODSs carrying these fragments of thegraphics Object appear in the DS in sequence. The last_in_sequence_flagfield indicates an end of the graphics Object. Though the above ODS datastructure is based on a method of storing fragments in consecutive PESpackets without a gap, the fragments may instead be stored in PESpackets so as to leave some gaps in the PES packets.

The PDS is a functional Segment for defining a Palette used for colorconversion. The Palette is data showing combinations of Pixel Codes of 1to 255 and pixel values. A pixel value referred to here is made up of ared color difference component (Cr value), a blue color differencecomponent (Cb value), a luminance component (Y value), and atransparency (T value). Substituting a Pixel Code of each piece ofrun-length data into a pixel value on the Palette produces a color. FIG.7B shows a data structure of the PDS. As shown in the drawing, the PDSincludes a segment_type field showing a Segment type “PDS”, asegment_length field showing a data length of the PDS, a palette_idfield uniquely identifying the Palette, a palette_version_number fieldshowing a version of the PDS within the Epoch, and a palette_entry fieldcarrying information for each entry. The palette_entry field shows a redcolor difference component (Cr_value), a blue color difference component(Cb_value), a luminance component (Y_value), and a transparency(T_value) for each entry.

The WDS is a functional Segment for defining a rectangular area on theGraphics Plane. As mentioned earlier, memory management is continuouswithin an Epoch during which clearing and rendering are performed in afixed rectangular area on the Graphics Plane. This rectangular area onthe Graphics Plane is called a Window, which is defined by the WDS. FIG.8A shows a data structure of the WDS. As shown in the drawing, the WDSincludes a window_id field uniquely identifying the Window on theGraphics Plane, a window_horizontal_position field specifying ahorizontal position of a top left pixel of the Window on the GraphicsPlane, a window_vertical_position field specifying a vertical positionof the top left pixel of the Window on the Graphics Plane, awindow_width field specifying a width of the Window on the GraphicsPlane, and a window_height field specifying a height of the Window onthe Graphics Plane.

The window_horizontal_position field, the window_vertical_positionfield, the window_width field, and the window_height field can take thefollowing values. The Graphics Plane serves as a coordinate system forthese field values. This Graphics Plane has a two-dimensional sizedefined by video_height and video_width parameters.

The window_horizontal_position field specifies the horizontal positionof the top left pixel of the Window on the Graphics Plane, andaccordingly takes a value in a range of 0 to (video_width)−1. Thewindow_vertical_position field specifies the vertical position of thetop left pixel of the Window on the Graphics Plane, and accordinglytakes a value in a range of 0 to (video_height)−1.

The window_width field specifies the width of the Window on the GraphicsPlane, and accordingly takes a value in a range of 1 to(video_width)−(window_horizontal_position). The window_height fieldspecifies the height of the Window on the Graphics Plane, andaccordingly takes a value in a range of 1 to(video_height)−(window_vertical_position).

A position and size of a Window can be defined for each Epoch, usingthese window_horizontal_position, window_vertical_position,window_width, and window_height fields in the WDS. This makes itpossible for the author to adjust, at the time of authoring, a Window toappear in a desired margin of each picture in an Epoch so as not tointerfere with a pattern of the picture. Graphics for subtitlesdisplayed in this way can be viewed clearly. The WDS can be defined foreach Epoch. Accordingly, when the pattern of the picture changes withtime, graphics can be moved based on such a change so as not to decreasevisibility. This enhances the quality of the film to the same level asin the case where subtitles are integrated in a moving picture.

The following explains the END. The END is a functional Segmentindicating that the transmission of the DS is complete. The END ispositioned immediately after the last ODS in the DS. The END includes asegment_type field showing a Segment type “END” and a segment_lengthfield showing a data length of the END. These fields are not mainfeatures of the present invention and therefore their explanation hasbeen omitted.

The following explains the PCS (Composition Segment).

The PCS is a functional Segment for composing a screen that can besynchronized with a moving picture. FIG. 8B shows a data structure ofthe PCS. As shown in the drawing, the PCS includes a segment_type field,a segment_length field, a composition_number field, a composition_statefield, a palette_update_flag field, a palette_id_ref field, andcomposition_object(1) to composition_object(m) fields.

The composition_number field uniquely identifies a graphics update inthe DS, using a number from 0 to 15. In more detail, thecomposition_number field is incremented by 1 for each graphics updatefrom the beginning of the Epoch to the PCS.

The composition_state field indicates whether the DS is a Normal CaseDS, an Acquisition Point DS, or an Epoch Start DS.

The palette_update_flag field shows whether the PCS describes aPalette-only Display Update. The Palette-only Display Update refers tosuch an update that only replaces a previous Palette with a new Palette.To indicate a Palette-only Display Update, the palette_update_flag fieldis set to 1.

The palette_id_ref field specifies the Palette to be used in the DS.

The composition_object(1) to composition_object(m) fields each containcontrol information for realizing a screen composition in the DS. InFIG. BB, dashed lines wd1 indicate an internal structure ofcomposition_object(i) as one example. As illustrated,composition_object(i) includes an object_id_ref field, a window_id_reffield, an object_cropped_flag field, an object_horizontal_positionfield, an object_vertical_position field, and cropping_rectangleinformation(1) to cropping_rectangle information(n).

The object_id_ref field shows a reference value of an identifier of agraphics Object (object_id). This reference value specifies the graphicsObject that is to be used in order to produce a screen compositioncorresponding to composition_object(i).

The window_id_ref field shows a reference value of an identifier of aWindow (window_id). This reference value specifies the Window in whichthe graphics Object is to be displayed in order to produce the screencomposition corresponding to composition_object(i).

The object_cropped_flag field shows whether the graphics Object croppedin the Object Buffer is to be displayed or not. When theobject_cropped_flag field is set to 1, the graphics Object cropped inthe Object Buffer is displayed. When the object_cropped_flag field isset to 0, the graphics Object cropped in the Object Buffer is notdisplayed.

The object_horizontal_position field specifies a horizontal position ofa top left pixel of the graphics Object on the Graphics Plane.

The object_vertical_position field specifies a vertical position of thetop left pixel of the graphics Object on the Graphics Plane.

The cropping_rectangle information(1) to cropping_rectangleinformation(n) fields are valid when the object_cropped_flag field valueis 1. Dashed lines wd2 indicate an internal structure ofcropping_rectangle information(i) as one example. As illustrated,cropping_rectangle information(i) includes anobject_cropping_horizontal_position field, anobject_cropping_vertical_position field, an object_cropping_width field,and an object_cropping_height field.

The object_cropping_horizontal_position field specifies a horizontalposition of a top left corner of a cropping rectangle in the graphicsObject. The cropping rectangle is used for taking out one part of thegraphics Object, and corresponds to a “region” in ETSI EN 300 743.

The object_cropping_vertical_position field specifies a verticalposition of the top left corner of the cropping rectangle in thegraphics Object.

The object_cropping_width field specifies a horizontal length of thecropping rectangle in the graphics Object.

The object_cropping_height field specifies a vertical length of thecropping rectangle in the graphics Object.

The following explains a specific description of the PCS, using anexample where the three subtitles “Actually”, “I lied to you.”, and“Sorry.” shown in FIG. 6 are displayed sequentially by three operationsof writing to the Graphics Plane as the reproduction of the movingpicture progresses. FIG. 9 shows an example description for realizingsuch subtitling. In the drawing, an Epoch has DS1 (Epoch Start DS), DS2(Normal Case DS), and DS3 (Normal Case DS). DS1 includes a WDS defininga Window in which the subtitles are to be displayed, an ODS showing theline “Actually I lied to you. Sorry.”, and a PCS. DS2 includes a PCS.DS3 includes a PCS.

Each of these PCSs has the following description. FIGS. 10 to 12 showexample descriptions of the WDS and the PCSs belonging to DS1 to DS3.

FIG. 10 shows descriptions of the PCS and the WDS in DS1. In thedrawing, a window_horizontal_position field value and awindow_vertical_position field value in the WDS specify top leftcoordinates LP1 of the Window on the Graphics Plane, and a window_widthfield value and a window_height field value in the WDS specify a widthand height of the Window.

An object_cropping_horizontal_position field value and anobject_cropping_vertical_position field value of cropping_rectangleinformation in the PCS specify top left coordinates ST1 of a croppingrectangle in a coordinate system whose origin is top left coordinates ofthe graphics Object in the Object Buffer. The cropping rectangle is anarea (enclosed by a thick-line box) defined by an object_cropping_widthfield value and an object_cropping_height field value from top leftcoordinates ST1. A cropped graphics Object is positioned in area cp1(enclosed by a dashed-line box) so that a top left corner of the croppedgraphics Object lies at a pixel specified by anobject_horizontal_position field value and an object_vertical_positionfield value in the coordinate system of the Graphics Plane. In this way,the subtitle “Actually” out of “Actually I lied to you. Sorry.” iswritten into the Window on the Graphics Plane. The subtitle “Actually”is overlaid on a picture and a resultant image is displayed.

FIG. 11 shows a description of the PCS in DS2. Since the description ofthe WDS in the drawing is the same as that in FIG. 10, its explanationhas been omitted. Meanwhile, the description of cropping_rectangleinformation differs from that in FIG. 10. In FIG. 11, anobject_cropping_horizontal_position field value and anobject_cropping_vertical_position field value in cropping_rectangleinformation specify top left coordinates of a cropping rectanglecorresponding to the subtitle “I lied to you.” in the Object Buffer, andan object_cropping_height field value and an object_cropping_width fieldvalue specify a height and width of the cropping rectangle. As a result,the subtitle “I lied to you.” is written into the Window on the GraphicsPlane. The subtitle “I lied to you.” is overlaid on a picture and aresultant image is displayed.

FIG. 12 shows a description of the PCS in DS3. Since the description ofthe WDS in the drawing is the same as that in FIG. 10, its explanationhas been omitted. Meanwhile, the description of cropping_rectangleinformation differs from that in FIG. 10. In FIG. 12, anobject_cropping_horizontal_position field value and anobject_cropping_vertical_position field value specify top leftcoordinates of a cropping rectangle corresponding to the subtitle“Sorry.” in the Object Buffer, and an object_cropping_height field valueand an object_cropping_width field value specify a height and width ofthe cropping rectangle. As a result, the subtitle “Sorry.” is writteninto the Window on the Graphics Plane. The subtitle “Sorry.” is overlaidon a picture and a resultant image is displayed. FIG. 13 shows a memoryspace of the Object Buffer when performing graphics updates such asthose shown in FIG. 10 to 12. As illustrated, the Object Buffer has fourstorage areas A to D which each have a fixed height and width and afixed position. Of storage areas A to D, the subtitle shown in FIG. 10is stored in storage area A. Each of storage areas A to D is identifiedby an object_id corresponding to a graphics Object to be stored in thatstorage area. In detail, storage area A is identified by object_id=1,storage area B is identified by object_id=2, storage area C isidentified by object_id=3, and storage area D is identified byobject_id=4. To maintain an efficiency of transfer from the ObjectBuffer to the Graphics Plane, the height and width of each of storageareas A to D are fixed. This being so, when a graphics Object havingsome object_id is obtained as a result of decoding, that graphics Objectis written in a storage area identified by the object_id over anexisting graphics Object. For example, to display a subtitle in the sameposition and size as the subtitles displayed in FIGS. 10 to 12, an ODShaving the same object_id as the ODS in DS1 needs to be provided in asucceeding DS. By adding the same object_id in such a way, a graphicsObject in the Object Buffer is overwritten by a new graphics Object,which is displayed in the same position and size as the overwrittengraphics Object.

The following explains constraints for achieving display effects. Todisplay subtitles smoothly, it is necessary to perform clearing andrendering on a Window. When performing Window clearing and Windowrendering at a frame rate of video frames, the following rate oftransfer from the Object Buffer to the Graphics Plane is required.

First, a constraint on the size of the Window is examined. Let Rc be thetransfer rate from the Object Buffer to the Graphics Plane. In aworst-case scenario, the Window clearing and the Window rendering needto be performed at Rc. In other words, each of the Window clearing andthe Window rendering needs to be performed at half of Rc (Rc/2).

To synchronize the Window clearing and the Window rendering with a videoframe,(Window size)×(frame rate)≈Rc/2

needs to be satisfied. If the frame rate is 29.97,Rc=(Window size)×2×29.97

To display a subtitle, the size of the Window needs to be at least about25% to 33% of the entire Graphics Plane. If a total number of pixels ofthe Graphics Plane is 1920×1080 and a bit length of an index per pixelis 8 bits, a total capacity of the Graphics Plane is 2 Mbytes(≈1920×1080×8).

Suppose the size of the Window is ¼ of the Graphics Plane, i.e., 500Kbytes (=2 Mbytes/4). Substituting this to the above formula yieldsRc=256 Mbps (500 Kbytes×2×29.97).

Thus, if the size of the Window is about 25% to 33% of the GraphicsPlane, display effects of subtitles can be achieved without losingsynchronization with a moving picture, so long as the subtitles aredisplayed with Rc=256 Mbps.

If the Window clearing and the Window rendering may be performed at ½ or¼ of the video frame rate, the size of the Window can be doubled orquadrupled with the same Rc. The following explains a position and rangeof a Window. As mentioned earlier, a position and range of a Window arefixed within an Epoch, for the following reason.

If the position or range of the Window varies in the Epoch, a writeaddress to the Graphics Plane needs to be changed. This incurs overhead,which causes a drop in transfer rate Rc from the Object Buffer to theGraphics Plane.

A number of graphics Objects that can be displayed simultaneously in oneWindow is limited, in order to reduce overhead when transferring decodedgraphics Objects to the Graphics Plane. The overhead mentioned hereoccurs when setting addresses of edge parts of the graphics Objects.This overhead increases if the number of edge parts is greater.

If there is no limitation on the number of graphics Objects that can bedisplayed in one Window, the overhead occurs unlimitedly whentransferring graphics objects to the Graphics Plane, which increases avariation in transfer load. On the other hand, if the number of graphicsObjects in one Window is limited to 2, transfer rate Rc can be set on anassumption that the number of instances of overhead is 4 at the worst.Hence a minimum standard for transfer rate Rc can be determined easily.This completes the explanation on a Window.

The following explains how DSs carrying functional Segments such as PCSsand ODSs described above are allocated on the reproduction time axis ofthe AV Clip. An Epoch is a time period on the reproduction time axisduring which memory management is continuous, and is made up of one ormore DSs. Hence it is important to effectively allocate DSs on thereproduction time axis of the AV Clip. The reproduction time axis of theAV Clip mentioned here is a time axis for defining decoding times andpresentation times of individual pictures which constitute the videostream multiplexed in the AV Clip. Decoding times and presentation timeson the reproduction time axis are expressed with a time accuracy of 90KHz. DTSs and PTSs of PCSs and ODSs in DSs specify timings forsynchronous control on this reproduction time axis. In other words, theDSs are allocated on the reproduction time axis by exercisingsynchronous control using the DTSs and PTSs of the PCSs and ODSs.

Synchronous control exercised using a DTS and a PTS of an ODS isexplained first.

The DTS shows a time at which a decoding process of the ODS is to bestarted, with an accuracy of 90 KHz. The PTS shows a time at which thedecoding process of the ODS is to be completed, with an accuracy of 90KHz.

The decoding process is made up of decoding the ODS and transferring anuncompressed graphics Object generated by the decoding to the ObjectBuffer. This decoding process does not complete instantaneously, butrequires a certain length of time. The DTS and the PTS of the ODSrespectively show the decoding start time and the decoding end time ofthe ODS, to specify the beginning and end of the decoding process.

Since the time shown by the PTS is a deadline, it is necessary to decodethe ODS and store an uncompressed graphics Object in the Object Bufferby the time shown by the PTS.

A decoding start time of arbitrary ODSj in DSn is specified byDTS(DSn[ODSj]) with an accuracy of 90 KHz. This being so, a decoding endtime of ODSj in DSn (i.e. PTS(DSn[ODSj]) is a sum of DTS(DSn[ODSj]) anda maximum time required for a decoding process.

Let SIZE(DSn[ODSj]) denote a size of ODSj, and Rd denote an ODS decodingrate. Then the maximum time required for the decoding process (inseconds) is SIZE(DSn[ODSj])//Rd. The symbol “//” represents an operatorfor a division with a fractional part being rounded up.

By converting this maximum time to the accuracy of 90 KHz and adding theresult to the DTS of ODSj, the decoding end time of ODSj specified bythe PTS is calculated with the accuracy of 90 KHz.

This PTS of ODSj in DSn can be expressed by the following formula:PTS(DSn[ODSj])=DTS(DSn[ODSj])+90,000×(SIZE(DSn[ODSj])//Rd)

Also, two adjacent ODSs (ODSj and ODSj+1) in DSn need to satisfy thefollowing relationship:PTS(DSn[ODSj])≦DTS(DSn[ODSj+1])

An END in DSn indicates an end of DSn. Therefore, the END shows adecoding end time of a last ODS (ODSlast) in DSn. The decoding end timeof ODSlast is shown by a PTS of ODSlast (PTS(DSn[ODSlast])), so that aPTS of the END is set as follows:PTS(DSn[END])=PTS(DSn[ODSlast])

Meanwhile, a DTS and a PTS of a PCS in DSn are set in the followingmanner.

The DTS of the PCS shows either a decoding start time of a top ODS(ODS1) in DSn or a time earlier than that. This is because the PCS needsto be loaded in a buffer of the reproduction apparatus 200 at the sametime as or earlier than the decoding start time of ODS1 (DTS(DSn[ODS1]))and a time at which a top PDS (PDS1) in DSn becomes valid(PTS(DSn[PDS1])). Which is to say, the DTS of the PCS needs to satisfythe following formulas:DTS(DSn[PCS])≦DTS(DSn[ODS1])DTS(DSn[PCS])≦PTS(DSn[PDS1])

On the other hand, the PTS of the PCS is calculated as follows:PTS(DSn[PCS])≧DTS(DSn[PCS])+DECODEDURATION(DSn)

Here, DECODEDURATION(DSn) indicates a time required for decoding andpresenting all graphics Objects used for updates described in the PCS inDSn. Though DECODEDURATION(DSn) is not a fixed value, it will not beaffected by factors such as differences in state or implementation ofreproduction apparatuses. When a graphics Object used for a screencomposition described by the PCS in DSn is denoted by DSn.PCS.OBJ[j],DECODEDURATION(DSn) is varied by (i) a time required for Windowclearing, (ii) a time required for decoding DSn.PCS.OBJ[j], and (iii) atime required for writing DSn.PCS.OBJ[j] on the Graphics Plane.Accordingly, DECODEDURATION(DSn) is the same regardless ofimplementations of reproduction apparatuses, so long as Rd and Rc arepredetermined. Therefore, the length of each of the above time periodsis calculated to specify the PTS of the PCS, at the time of authoring.

The calculation of DECODEDURATION(DSn) is carried out based on a programshown in FIG. 14. FIG. 15 and FIGS. 16A and 16B are flowcharts showingan algorithm of this program. A procedure of calculatingDECODEDURATION(DSn) is explained below, with reference to thesedrawings. In FIG. 15, a PLANEINITIALIZATIONTIME function is called, anda return value is added to decode_duration (S1). ThePLANEINITIALIZATIONTIME function (FIG. 16A) is a function forcalculating a time required for initializing the Graphics Plane in orderto produce display for DSn. In step S1, this PLANEINITIALIZATIONTIMEfunction is called using DSn, DSn.PCS.OBJ[0], and decode_duration asarguments.

FIG. 16A shows a procedure of the PLANEINITIALIZATIONTIME function. Inthe drawing, initialize_duration is a variable indicating a return valueof the PLANEINITIALIZATIONTIME function.

Step S2 judges whether a composition_state field of the PCS in DSn showsEpoch Start. If the composition_state field shows Epoch Start (S2:YES,DSn.PCS.composition_state==EPOCH_START in FIG. 14), a time required forclearing the Graphics Plane is set as initialize_duration (S3).

Suppose transfer rate Rc from the Object Buffer to the Graphics Plane is256,000,000 and a total size of the Graphics Plane is(video_width)*(video_height), as mentioned above. Then the time requiredfor clearing the Graphics Plane (in seconds) is(video_width)*(video_height)//256,000,000. This is multiplied by 90,000Hz, to express in the PTS accuracy. Hence the time required for clearingthe Graphics Plane is 90,000×(video_width)*(video_height)//256,000,000.This is added to initialize_duration, which is returned as a returnvalue.

If the composition_state field does not show Epoch Start (S2:NO), anoperation of adding a time required for clearing Window[i] toinitialize_duration is carried out for all Windows[i] (S4). Supposetransfer rate Rc from the Object Buffer to the Graphics Plane is256,000,000 as mentioned earlier, and a total size of Windows[i] isΣSIZE(WDS.WIN[i]). Then a time required for clearing all Windows[i] (inseconds) is ΣSIZE(WDS.WIN[i])//256,000,000. This is multiplied by 90,000Hz to express in the PTS accuracy. Hence the time required for clearingall Windows[i] is 90,000×ΣSIZE(WDS.WIN[i])//256,000,000. This is addedto initialize_duration, which is returned as a return value. Thiscompletes the PLANEINITIALIZATIONTIME function.

Referring back to FIG. 15, step S5 judges whether a number of graphicsObjects in DSn is 1 or 2 (if(DSn.PCS.num_of_objects==2,if(DSn.PCS.num_of_objects==1 in FIG. 14). If the number of graphicsObjects in DSn is 1 (S5:=1), a wait time for decoding that graphicsObject to complete is added to decode_duration (S6). The wait time iscalculated by calling a WAIT function (decode_duration+=WAIT(DSn,DSn.PCS.OBJ[0], decode_duration) in FIG. 14). The WAIT function iscalled using DSn, DSn.PCS.OBJ[0], and decode_duration as arguments, andwait_duration showing the wait time is returned as a return value.

FIG. 16B shows a procedure of the WAIT function.

In the WAIT function, current_duration is a variable to whichdecode_duration is set, and object_definition_ready_time is a variableindicating a PTS of graphics Object OBJ[i] in DSn.

Also, current_time is a variable indicating a sum of current_durationand a DTS of the PCS in DSn. If object_definition_ready_time is greaterthan current_time (S7:YES, if(current_time<object_definition_ready_time)in FIG. 14), a difference between object_definition_ready_time andcurrent_time is set to wait_duration, which is returned as a returnvalue (S8, wait_duration+=object_definition_ready_time−current_time inFIG. 14). This completes the WAIT function.

Referring back to FIG. 15, a sum of the return value of the WAITfunction and a time required for rendering on a Window to which OBJ[0]belongs (90,000*(SIZE(DSn.WDS.WIN[0]))//256,000,000) is set todecode_duration (S9).

The above procedure relates to the case when the number of graphicsObjects in DSn is 1. If the number of graphics Objects is 2 (S5:=2,if(DSn.PCS.num_of_objects==2) in FIG. 14), the WAIT function is calledusing DSn, DSn.PCS.OBJ[0], and decode_duration as arguments, and areturn value of the WAIT function is added to decode_duration (S10).

Step S11 judges whether the Window to which OBJ[0] belongs is the sameas a Window to which OBJ[1] belongs(if(DSn.PCS.OBJ[0].window_id==DSn.PCS.OBJ[1].window_id) in FIG. 14). Ifthe judgement is in the affirmative (S11:YES), the WAIT function iscalled using DSn, DSn.PCS.OBJ[1], and decode_duration as arguments, anda return value of the WAIT function is added to decode_duration (S12).Furthermore, a time required for rendering on the Window to which OBJ[0]and OBJ[1] belong (90,000*(SIZE(DSn.WDS.OBJ[0].window_id)//256,000,000)is added to decode_duration (S13).

If the judgement is in the negative (S11:NO), on the other hand, thetime required for rendering on the Window to which OBJ[0] belongs(90,000*(SIZE(DSn.WDS.OBJ[0].window_id)//256,000,000) is added todecode_duration (S15). After this, the WAIT function is called usingDSn, DSn.PCS.OBJ[1], and decode_duration as arguments, and a returnvalue of the WAIT function is added to decode_duration (Sl6).Furthermore, a time required for rendering on the Window to which OBJ[1]belongs (90,000*(SIZE(DSn.WDS.OBJ[l].window_id)//256,000,000) is addedto decode_duration (S17). In this way, DECODEDURATION(DSn) iscalculated.

The following explains how a PTS of a PCS in one DS is set, usingspecific examples.

FIG. 17A shows a situation where one OBJ (OBJ1) corresponding to one ODS(ODS1) belongs to one Window. FIGS. 17B and 17C are timing chartsshowing a relationship between parameters used in FIG. 14. Each of thesetiming charts has three levels. Of the three levels, the “Graphics Planeaccess” level and the “ODS decode” level indicate two processes whichare performed in parallel when reproducing the ODS. The above algorithmis based on an assumption that these two processes are performed inparallel.

Graphics Plane access is made up of clear period (1) and write period(3). Clear period (1) indicates either a time required for clearing theentire Graphics Plane (90,000×((size of the GraphicsPlane)//256,000,000)) or a time required for clearing all Windows on theGraphics Plane (Σ(90,000×((size of Window[i])//256,000,000))).

Write period (3) indicates a time required for rendering on the entireWindow (90,000×((size of the Window)//256,000,000)).

ODS decode is made up of decode period (2). Decode period (2) indicatesa time period from a DTS to a PTS of ODS1.

Clear period (1), decode period (2), and write period (3) can varydepending on the range to be cleared, the size of an ODS to be decoded,and the size of a graphics Object to be written to the Graphics Plane.In FIG. 17, the beginning of decode period (2) is assumed to be the sameas the beginning of clear period (1), for simplicity's sake.

FIG. 17B shows a case where decode period (2) is longer than clearperiod (1). In this case, decode_duration is a sum of decode period (2)and write period (3).

FIG. 17C shows a case where clear period (1) is longer than decodeperiod (2). In this case, decode_duration is a sum of clear period (1)and write period (3).

FIGS. 18A to 18C show a situation where two OBJs (OBJ1 and OBJ2)corresponding to two ODSs (ODS1 and ODS2) belong to one Window. In FIGS.18B and 18C, decode period (2) indicates a total time required fordecoding ODS1 and ODS2. Likewise, write period (3) indicates a totaltime required for writing OBJ1 and OBJ2 to the Graphics Plane. Thoughthe number of ODSs is two, decode_duration can be calculated in the sameway as in FIG. 17. In detail, if decode period (2) of ODS1 and ODS2 islonger than clear period (1), decode_duration is a sum of decode period(2) and write period (3) as shown in FIG. 18B.

If clear period (1) is longer than decode period (2), decode_duration isa sum of clear period (1) and write period (3) as shown in FIG. 18C.

FIGS. 19A to 19C show a situation where OBJ1 belongs to Window1 and OBJ2belongs to Window2. In this case too, if clear period (1) is longer thandecode period (2) of ODS1 and ODS2, decode_duration is a sum of clearperiod (1) and write period (3). If clear period (1) is shorter thandecode period (2), on the other hand, OBJ1 can be written to Window1without waiting for the end of decode period (2). In such a case,decode_duration is not simply a sum of decode period (2) and writeperiod (3). Let write period (31) denote a time required for writingOBJ1 to Window1 and write period (32) denote a time required for writingOBJ2 to Window2. FIG. 19B shows a case where decode period (2) is longerthan a sum of clear period (1) and write period (31). In this case,decode_duration is a sum of decode period (2) and write period (32).

FIG. 19C shows a case where a sum of clear period (1) and write period(31) is longer than decode period (2). In this case, decode_duration isa sum of clear period (1), write period (31), and write period (32).

The size of the Graphics Plane is fixed according to a player model.Also, sizes and numbers of Windows and ODSs are set in advance at thetime of authoring. Hence decode_duration can be calculated as one of thesum of clear period (1) and write period (3), the sum of decode period(2) and write period (3), the sum of decode period (2) and write period(32), and the sum of clear period (1), write period (31), and writeperiod (32). By setting the PTS of the PCS based on such calculateddecode_duration, graphics can be synchronized with picture data withhigh accuracy. Such accurate synchronous control is achieved by definingWindows and restricting clearing and rendering operations within theWindows. Thus, the introduction of the concept “Window” in authoring isof great significance.

The following explains how a DTS and a PTS of the WDS in DSn are set.The DTS of the WDS is set so as to satisfy the following formula:DTS(DSn[WDS])≧DTS(DSn[PCS])

The PTS of the WDS specifies a deadline for starting writing to theGraphics Plane. Since writing to the Graphics Plane is restricted to aWindow, the time to start writing to the Graphics Plane can bedetermined by subtracting a time required for rendering on all Windowsfrom the time shown by the PTS of the PCS. Let ΣSIZE(WDS.WIN[i]) be atotal size of Windows[i]. Then a time required for clearing andrendering on all Windows[i] is ΣSIZE(WDS.WIN[i])//256,000,000.Expressing this time with the accuracy of 90,000 KHz yields90,000×ΣSIZE(WDS.WIN[i])//256,000,000.

Accordingly, the PTS of the WDS can be calculated as follows:PTS(DSn[WDS])=PTS(DSn[PCS])−90,000×ΣSIZE(WDS.WIN[i])//256,000,000

Since the PTS of the WDS shows the deadline, the writing to the GraphicsPlane can be launched earlier than the time shown by this PTS. Which isto say, once decoding of one ODS belonging to one of two Windows hascompleted, a graphics Object obtained by the decoding can be immediatelywritten to the Window as shown in FIG. 19.

Thus, a Window can be allocated to a desired point on the reproductiontime axis of the AV Clip, using the DTS and the PTS of the WDS. Thiscompletes the explanation on the DTS and PTS of each of the PCS and theWDS in DSn.

The following explains a forward reference by a PCS. A forward referenceis a type of reproduction control that references a graphics Objectwhich has already been stored in the reproduction apparatus 200. FIG. 20shows forward reference PCSs. The drawing shows three DSs, i.e. DSn,DSn+1, and DSn+2. PCSs of DSn+1 and DSn+2 are forward reference PCSs.Arrows yy1 and yy2 indicate forward references made by these PCSs.Graphics Objects defined in ODS#1 to ODS#v in DSn are forward-referencedby these PCSs. Here, DSn contains a plurality of graphics Objects, i.e.ODS#1 to ODS#u and ODS#u+1 to ODS#v. Of these graphics Objects, ODS#1 toODS#u are referenced by a PCS of DSn, but ODS#u+1 to ODS#v are notreferenced by the PCS of DSn. These graphics Objects not referenced bythe PCS of DSn are forward-referenced by the PCSs of DSn+1 and DSn+2.

Such a graphics Object forward-referenced by a PCS in one DS has alreadybeen stored in the reproduction apparatus 200, so that there is no needto wait for decoding of an ODS. This being so, by feeding only a PCSinto the reproduction apparatus 200, display control based on that PCScan be carried out. Since the display control is carried out promptlywhen the PCS is fed into the reproduction apparatus 200, a graphicsupdate of replacing existing graphics with new graphics can be performedwith a short time interval. Repeating such a short-interval graphicsupdate makes it possible to realize display control of moving graphicssmoothly as the reproduction of the moving picture progresses. Twodescription examples of forward reference PCSs are given below.

A description example of forward reference PCSs for realizing a displayeffect of moving graphics on a screen is explained first. FIG. 21A showsan Epoch having such a display effect. This Epoch is made up of nine DSs(DS0 to DS8). DS0, the first DS of the Epoch, includes a PCS, a PDS, andan ODS. The ODS in DS0 defines a subtitle “My heart is fluttering” shownin FIG. 22A, and has object_id=1. Meanwhile, DS1 to DS8 which follow DS0each include only a PCS.

FIG. 21B shows the composition_object fields and PTSs of the PCSs in DS1to DS8. As illustrated, the PCSs in DS1 to DS8 designate the graphicsObject specified by object_id_ref=1 to be displayed at respectivecoordinates (x1,y1), (x2,y2), (x3,y3), . . . , (x8,y8) specified by theobject_horizontal_position and object_vertical_position fields. FIG. 22Bshows positions of coordinates (x1,y1), (x2,y2), (x3,y3), . . . ,(x8,y8) in a coordinate system defined by a Window. As can be understoodfrom the drawing, coordinates (x1,y1), (x2,y2), (x3,y3), . . . , (x8,y8)lie on a wavy line. Coordinates (x1,y1), (x2,y2), (x3,y3), . . . ,(x8,y8) are set so that the graphics Object stays within the Windowdefined by the WDS. This is because the synchronization between thegraphics display of the graphics stream and the video display of thevideo stream cannot be established if the graphics Object is notcontained within the Window.

In FIG. 21B, the PTSs of the PCSs in DS1 to DS8 designate times t1, t2,t3, . . . , t8, on the reproduction time axis of the video stream, atwhich the graphics Object is to be displayed at respective coordinates(x1,y1), (x2,y2), (x3,y3), . . . , (x8,y8).

FIG. 23 shows a display effect when DS0 to DS8 having the above PCSs aresequentially fed into the reproduction apparatus 200. At t1 on thereproduction time axis of the video stream, DS1 is fed into thereproduction apparatus 200, and as a result the graphics is displayed atcoordinates (x1,y1) within the Window. At t4, DS4 is fed into thereproduction apparatus 200, and as a result the graphics is displayed atcoordinates (x4,y4). At t6, DS6 is fed into the reproduction apparatus200, and as a result the graphics is displayed at coordinates (x6,y6).At t8, DS8 is fed into the reproduction apparatus 200, and as a resultthe graphics is displayed at coordinates (x8,y8). Since thesecoordinates (x1,y1), (x4,y4), (x6,y6), and (x8,y8) lie on a wavy line,the graphics will end up tracing the wavy line as shown in FIG. 24.

Thus, when such DSs including PCSs which each have specifiedobject_horizontal_position and object_vertical_position field values aresequentially fed into the reproduction apparatus 200, the reproductionapparatus 200 displays the graphics Object already stored in the ObjectBuffer in accordance with these PCSs. Since the position of the graphicscan be changed just by feeding the control information into thereproduction apparatus 200, quick movements of graphics on the screencan be achieved.

When feeding such DSs into the reproduction apparatus 200, a new PCS maybe written over an existing PCS. This allows the reproduction apparatus200 to hold only a newest PCS, even if there are tens or hundreds ofPCSs showing display coordinates of graphics for smooth movement. Inthis way, the memory occupancy in the reproduction apparatus 200 can beminimized.

A Palette-only Display Update is explained next, as another descriptionexample of forward reference PCSs. FIG. 25 shows a series of DSs forrealizing a Palette-only Display Update. In the drawing, the first levelshows the series of DSs (DS0, DS1, DS2, and DS3) for the Palette-onlyDisplay Update. DS0 has a PCS, four PDSs (PDS0, PDS1, PDS2, and PDS3),and an ODS. DS2, DS3, and DS4 that follow DS0 each have only a PCS.

The ODS in DS0 defines a subtitle “I will never forget”, as indicated byguide lines hv1. The words “I”, “will”, “never”, and “forget” are eachconstituted by a plurality of pieces of run-length data (indicated byrectangles shown in FIG. 25). Run-length data constituting “I” has PixelCode 0. Run-length data constituting “will” has Pixel Code 1. Run-lengthdata constituting “never” has Pixel Code 2. Run-length data constituting“forget” has Pixel Code 3.

FIG. 26A shows the four PDSs in DS0.

PDS0 which is the first PDS in DS0 (on the first level) haspalette_id=0, and assigns a red color to Pixel Code 0 and a white colorto Pixel Codes 1 to 3.

PDS1 which is the second PDS in DS0 (on the second level) haspalette_id=1, and assigns the red color to Pixel Codes 0 and 1 and thewhite color to Pixel Codes 2 and 3.

PDS2 which is the third PDS in DS0 (on the third level) haspalette_id=2, and assigns the red color to Pixel Codes 0, 1, and 2 andthe white color to Pixel Code 3.

PDS3 which is the fourth PDS in DS0 (on the fourth level) haspalette_id=3, and assigns the red color to Pixel Codes 0 to 3.

FIG. 26B show descriptions of the PCSs in DS0 to DS3.

The PCS in DS0 (on the first level) designates display control to beperformed using a graphics Object specified by object_id_ref=1. This PCShas palette_update_flag=0 and palette_id_ref=0. Accordingly, when DS0 isfed into the reproduction apparatus 200, the graphics Object shown bythe ODS in DS0 is displayed using the PDS specified by palette_id=0.

The PCS in DS1 (on the second level) designates display control to beperformed using the graphics Object specified by object_id_ref=1. ThisPCS has palette_update_flag=1 and palette_id_ref=1. Accordingly, whenDS1 is fed into the reproduction apparatus 200, the display of thegraphics Object is updated using the PDS specified by palette_id=1.

The PCS in DS2 (on the third level) designates display control to beperformed using the graphics Object specified by object_id_ref=1. ThisPCS has palette_update_flag=1 and palette_id_ref=2. Accordingly, whenDS2 is fed into the reproduction apparatus 200, the display of thegraphics Object is updated using the PDS specified by palette_id=2.

The PCS in DS3 (on the fourth level) designates display control to beperformed using the graphics Object specified by object_id_ref=1. ThisPCS has palette_update_flag=1 and palette_id_ref=3. Accordingly, whenDS3 is fed into the reproduction apparatus 200, the display of thegraphics Object is updated using the PDS specified by palette_id=3. Bysetting the PCSs in DS0 to DS3 in this way, it is possible to repeatedlyupdate the graphics Object belonging to DS0 using different palettedata.

FIG. 27 shows a display effect achieved when these four DSs aresequentially fed into the reproduction apparatus 200. When DS0 is fedinto the reproduction apparatus 200, the subtitle “I will never forget”is displayed with the word “I” being colored in red. When DS1 is fedinto the reproduction apparatus 200, the word “will” turns red. When DS2is fed into the reproduction apparatus 200, the word “never” turns red.When DS3 is fed into the reproduction apparatus 200, the word “forget”turns red, as a result of which the entire subtitle “I will neverforget” is displayed in red. Thus, the words constituting the subtitle“I will never forget” change from white to red one by one beginning withthe leftmost word, as the DSs are sequentially fed from the BD-ROM 100to the reproduction apparatus 200. This produces a display effect ofchanging a subtitle in color as the reproduction of the audio streamprogresses.

Note that the data structures of DSs (PCS, WDS, PDS, and ODS) explainedabove are instances of class structures written in a programminglanguage. The author writes the class structures according to the syntaxdefined in the Blu-ray Disc Read-Only Format, to create these datastructures on the BD-ROM 100. This completes the explanation on therecording medium according to the first embodiment of the presentinvention. The following explains a reproduction apparatus according tothe first embodiment of the present invention. FIG. 28 shows an internalconstruction of the reproduction apparatus 200. The reproductionapparatus 200 is manufactured based on this internal construction. Thereproduction apparatus 200 is roughly made up of three parts that are asystem LSI, a drive device, and a microcomputer system. The reproductionapparatus 200 can be manufactured by mounting these parts on a cabinetand substrate of the apparatus. The system LSI is an integrated circuitincluding various processing units for achieving the functions of thereproduction apparatus 200. The reproduction apparatus 200 includes a BDdrivel, a Read Buffer 2, a PID filter 3, Transport Buffers 4 a, 4 b, and4 c, a peripheral circuit 4 d, a Video Decoder 5, a Video Plane 6, anAudio Decoder 7, a Graphics Plane 8, a CLUT unit 9, an adder 10, and aGraphics Decoder 12. The Graphics Decoder 12 includes a Coded DataBuffer 13, a peripheral circuit 13 a, a Stream Graphics Processor 14, anObject Buffer 15, a Composition Buffer 16, and a Graphics Controller 17.

The BD drive 1 performs loading, reading, and ejecting of the BD-ROM100. The BD drive 1 accesses to the BD-ROM 100.

The Read Buffer 2 is a FIFO (first-in first-out) memory. Accordingly, TSpackets read from the BD-ROM 100 are removed from the Read Buffer 2 inthe same order as they arrive.

The PID filter 3 performs filtering on TS packets output from the ReadBuffer 2. In more detail, the PID filter 3 passes only TS packets havingpredetermined PIDs to the Transport Buffers 4 a, 4 b, and 4 c. There isno buffering inside the PID filter 3. Accordingly, TS packets enteringthe PID filter 3 are instantaneously written to the Transport Buffers 4a, 4 b, and 4 c.

The Transport Buffers 4 a, 4 b, and 4 c are FIFO memories for storing TSpackets output from the PID filter 3. A speed at which a TS packet isread from the Transport Buffer 4 a is denoted by transfer rate Rx.

The peripheral circuit 4 d has a wired logic for converting TS packetsread from the Transport Buffer 4 a to functional Segments. Thefunctional Segments are then stored in the Coded Data Buffer 13.

The Video Decoder 5 decodes TS packets output from the PID filter 3 toobtain uncompressed pictures, and writes them to the Video Plane 6.

The Video Plane 6 is a plane memory for a moving picture.

The Audio Decoder 7 decodes TS packets output from the PID filter 3, andoutputs uncompressed audio data.

The Graphics Plane 8 is a plane memory having a memory area of onescreen, and is capable of storing uncompressed graphics of one screen.

The CLUT unit 9 converts index colors of the uncompressed graphics onthe Graphics Plane 8, based on Y, Cr, and Cb values shown in a PDS.

The adder 10 multiplies the uncompressed graphics converted by the CLUTunit 9, by a T value (transparency) shown in the PDS. The adder 10 thenperforms addition for corresponding pixels in the resulting uncompressedgraphics and the uncompressed picture data on the Video Plane 6, andoutputs a resultant image.

The Graphics Decoder 12 decodes a graphics stream to obtain uncompressedgraphics, and writes the uncompressed graphics to the Graphics Plane 8as graphics Objects. As a result of decoding the graphics stream,subtitles and menus appear on the screen.

This Graphics Decoder 12 executes pipeline processing, by reading agraphics Object belonging to DSn from the Object Buffer 15 whilstsimultaneously writing a graphics Object belonging to DSn+1 to theObject Buffer 15.

The Graphics Decoder 12 includes the Coded Data Buffer 13, theperipheral circuit 13 a, the Stream Graphics Processor 14, the ObjectBuffer 15, the Composition Buffer 16, and the Graphics Controller 17.

The Coded Data Buffer 13 is used for storing functional Segmentstogether with DTSs and PTSs. Such functional Segments are obtained byremoving a TS packet header and a PES packet header from each TS packetstored in the Transport Buffer 4 a and arranging remaining payloads insequence. DTSs and PTSs contained in the removed TS packet headers andPES packet headers are stored in the Coded Data Buffer 13 incorrespondence with the functional Segments.

The peripheral circuit 13 a has a wired logic for transferring data fromthe Coded Data Buffer 13 to the Stream Graphics Processor 14 andtransferring data from the Coded Data Buffer 13 to the CompositionBuffer 16. In more detail, when the current time reaches a DTS of anODS, the peripheral circuit 13 a transfers the ODS from the Coded DataBuffer 13 to the Stream Graphics Processor 14. Also, when the currenttime reaches a DTS of a PCS or a PDS, the peripheral circuit 13 atransfers the PCS or the PDS from the Coded Data Buffer 13 to theComposition Buffer 16.

The Stream Graphics Processor 14 decodes the ODS to obtain uncompressedgraphics having index colors, and transfers the uncompressed graphics tothe Object Buffer 15 as a graphics Object. The decoding by the StreamGraphics Processor 14 is instantaneous, and the graphics Object obtainedby the decoding is temporarily stored in the Stream Graphics Processor14. Though the decoding by the Stream Graphics Processor 14 isinstantaneous, the transfer of the graphics Object from the StreamGraphics Processor 14 to the Object Buffer 15 is not instantaneous. Thisis because transfer to the Object Buffer 15 is performed at a transferrate of 128 Mbps in the player model of the Blu-ray Disc Read-OnlyFormat. An end of transfer of all graphics Objects belonging to a DS tothe Object Buffer 15 is shown by a PTS of an END in the DS. Therefore,processing of the next DS will not be started until the time shown bythe PTS of the END. Transfer of a graphics Object obtained by decodingeach ODS to the Object Buffer 15 starts at a time shown by a DTS of theODS and ends at a time shown by a PTS of the ODS.

If a graphics Object of DSn and a graphics Object of DSn+1 havedifferent object_ids, the Stream Graphics Processor 14 writes the twographics Objects in different storage areas of the Object Buffer 15.This allows pipeline presentation of the graphics Objects, without thegraphics Object of DSn being overwritten by the graphics Object ofDSn+1. If the graphics Object of DSn and the graphics Object of DSn+1have the same object_id, on the other hand, the Stream GraphicsProcessor 14 writes the graphics Object of DSn+1 to a storage area inthe Object Buffer 15 in which the graphics Object of DSn is stored, soas to overwrite the graphics Object of DSn. In this case, pipelineprocessing is not performed. Also, a DS may include ODSs which arereferenced by a PCS of the DS and ODSs which are not referenced by thePCS. The Stream Graphics Processor 14 sequentially decodes not only theODSs referenced by the PCS but also the ODSs not referenced by the PCS,and stores graphics obtained by the decoding to the Object Buffer 15.

The Object Buffer 15 corresponds to a pixel buffer in ETSI EN 300 743.Graphics Objects decoded by the Stream Graphics Processor 14 are storedin the Object Buffer 15. A size of the Object Buffer 15 needs to betwice or four times as large as that of the Graphics Plane 8. This isbecause the Object Buffer 15 needs to be capable of storing twice orfour times as much graphics as the Graphics Plane 8 in order to achievescrolling.

The Composition Buffer 16 is used for storing a PCS and a PDS. Whenactive periods of PCSs in DSn and DSn+1 overlap, the Composition Buffer16 stores the PCSs of both DSn and DSn+1.

The Graphics Controller 17 decodes the PCSs in the Composition Buffer16. Based on a decoding result, the Graphics Controller 17 writes agraphics Object of DSn+1 to the Object Buffer 15, while reading agraphics Object of DSn from the Object Buffer 15 and presenting it fordisplay. The presentation by the Graphics Controller 17 is performed ata time shown by a PTS of the PCS in DSn.

To produce a screen composition using a PCS, the Graphics Controller 17performs the following processing. The Graphics Controller 17 reads anobject_id_ref which is a reference value of an identifier of a graphicsObject, from the PCS. The Graphics Controller 17 takes out one part ofthe graphics Object that is defined by anobject_cropping_horizontal_position, anobject_cropping_vertical_position, an object_cropping_width, and anobject_cropping_height. The Graphics Controller 17 writes the taken parton the Graphics Plane 8 at a position shown by anobject_horizontal_position and an object_vertical_position. If thegraphics Object specified by the reference value belongs to the same DSas the PCS, the above writing to the Graphics Plane 8 cannot beperformed until after a corresponding ODS in that DS is decoded.

On the other hand, when a composition_state of the PCS shows NormalCase, the graphics Object has already been decoded and stored in theObject Buffer 15. Accordingly, the above writing to the Graphics Plane 8can be performed without waiting for decoding of an ODS.

To effect a Palette-only Display Update, the Graphics Controller 17performs the following processing. Once a DS has been read from theBD-ROM 100, the Graphics Controller 17 judges whether apalette_update_flag of a PCS in the DS is “1”. If thepalette_update_flag is “1”, the Graphics Controller 17 instructs theCLUT unit 9 to perform pixel value conversion using palette dataspecified by a palette_id of the PCS. The Graphics Controller 17 thenreads a graphics Object from the Object Buffer 15 and writes it to theGraphics Plane 8. Thus, a graphics update can be carried out just bysetting the palette data in the CLUT unit 9.

Recommended transfer rates and buffer sizes for realizing the PID filter3, the Transport Buffers 4 a, 4 b, and 4 c, the Graphics Plane 8, theCLUT unit 9, the Coded Data Buffer 13, the Stream Graphics Processor 14,the Object Buffer 15, the Composition Buffer 16, and the GraphicsController 17 are given below. FIG. 29 shows transfer rates Rx, Rc, andRd and sizes of the Graphics Plane 8, the Transport Buffer 4 a, theCoded Data Buffer 13, and the Object Buffer 15.

Transfer rate Rc (Pixel Composition Rate) from the Object Buffer 15 tothe Graphics Plane 8 is a highest transfer rate in the reproductionapparatus 200, and is calculated as 256 Mbps (=500 Kbytes×29.97×2) froma Window size and a frame rate.

Transfer rate Rd (Pixel Decoding Rate) from the Stream GraphicsProcessor 14 to the Object Buffer 15 does not need to coincide with theframe rate unlike Rc, and may be ½ or ¼ of Rc. Therefore, transfer rateRd is 128 Mbps or 64 Mbps.

Transfer rate Rx (Transport Buffer Leak Rate) from the Transport Buffer4 a to the Coded Data Buffer 13 is a transfer rate of ODSs in acompressed state. Accordingly, transfer rate Rx can be calculated bymultiplexing Rd by a compression rate of ODSs. For example, when thecompression rate is 25%, transfer rate Rx is 16 Mbps (=64 Mbps×25%).

These transfer rates and buffer sizes are merely shown as minimumstandards, and transfer rates and buffer sizes greater than those shownin FIG. 29 are equally applicable.

In the above constructed reproduction apparatus 200, the constructionelements perform processing in a pipeline.

FIG. 30 is a timing chart showing pipeline processing performed in thereproduction apparatus 200. In the drawing, the fifth level shows a DSon the BD-ROM 100. The fourth level shows write periods of a PCS, a WDS,a PDS, ODSs, and an END to the Coded Data Buffer 13. The third levelshows decode periods of the ODSs by the Stream Graphics Processor 14.The second level shows storage contents of the Composition Buffer 16.The first level shows operations of the Graphics Controller 17.

DTSs of ODS1 and ODS2 show t31 and t32 respectively. Therefore, ODS1 andODS2 need to be buffered in the Coded Data Buffer 13 by t31 and t32respectively. This being so, writing of ODS1 to the Coded Data Buffer 13is completed by t31 at which decode period dp1 begins, and writing ofODS2 to the Coded Data Buffer 13 is completed by t32 at which decodeperiod dp2 begins.

Meanwhile, PTSs of ODS1 and ODS2 show t32 and t33 respectively.Accordingly, decoding of ODS1 by the Stream Graphics Processor 14 iscompleted by t32, and decoding of ODS2 by the Stream Graphics Processor14 is completed by t33. Thus, an ODS is buffered in the Coded DataBuffer 13 by a time shown by a DTS of the ODS, and the buffered ODS isdecoded and transferred to the Object Buffer 15 by a time shown by a PTSof the ODS.

On the first level, cd1 denotes a time period needed for the GraphicsController 17 to clear the Graphics Plane 8, and td1 denotes a timeperiod needed for the Graphics Controller 17 to write graphics obtainedin the Object Buffer 15 to the Graphics Plane 8. A PTS of the WDS showsa deadline for starting writing the graphics. A PTS of the PCS shows atime at which the writing of the graphics to the Graphics Plane 8 endsand the written graphics is presented for display. Therefore,uncompressed graphics of one screen is obtained on the Graphics Plane 8at the time shown by the PTS of the PCS. The CLUT unit 9 performs colorconversion on the uncompressed graphics, and the adder 10 overlays thegraphics on an uncompressed picture stored on the Video Plane 6. Thisproduces a resultant image.

In the Graphics Decoder 12, the Stream Graphics Processor 14 continuesdecoding while the Graphics Controller 17 is clearing the Graphics Plane8. As a result of such pipeline processing, graphics can be displayedspeedily.

FIG. 30 shows an example when the clearing of the Graphics Plane 8 endsbefore the decoding of the ODSs. FIG. 31, on the other hand, is a timingchart showing pipeline processing when the decoding of the ODSs endsbefore the clearing of the Graphics Plane 8. In this case, uponcompletion of the decoding of the ODSs, the graphics obtained by thedecoding cannot yet be written to the Graphics Plane 8. Only after theclearing of the Graphics Plane 8 is completed, the graphics can bewritten to the Graphics Plane 8.

FIG. 32 is a timing chart showing changes in buffer occupancy in thereproduction apparatus 200. In the drawing, the first to fourth levelsshow changes in occupancy of the Graphics Plane 8, the Object Buffer 15,the Coded Data Buffer 13, and the Composition Buffer 16, respectively.These changes are shown in the form of a line graph in which thehorizontal axis represents time and the vertical axis representsoccupancy.

The fourth level shows changes in occupancy of the Composition Buffer16. As illustrated, the changes in occupancy of the Composition Buffer16 include monotone increase Vf0 with which the PCS output from theCoded Data Buffer 13 is stored.

The third level shows changes in occupancy of the Coded Data Buffer 13.As illustrated, the changes in occupancy of the Coded Data Buffer 13include monotone increases Vf1 and Vf2 with which ODS1 and ODS2 arestored, and monotone decreases Vg1 and Vg2 with which ODS1 and ODS2 aresequentially read by the Stream Graphics Processor 14. Slopes ofmonotone increases Vf1 and Vf2 are based on transfer rate Rx from theTransport Buffer 4 a to the Coded Data Buffer 13, whereas monotonedecreases Vg1 and Vg2 are instantaneous since decoding by the StreamGraphics Processor 14 is performed instantaneously. Which is to say, theStream Graphics Processor 14 decodes each ODS instantaneously and holdsuncompressed graphics obtained by the decoding. Since transfer rate Rdfrom the Stream Graphics Processor 14 to the Object Buffer 15 is 128Mbps, the occupancy of the Object Buffer 15 increases at 128 Mbps.

The second level shows changes in occupancy of the Object Buffer 15. Asillustrated, the changes in occupancy of the Object Buffer 15 includemonotone increases Vh1 and Vh2 with which the graphics Objects of ODS1and ODS2 output from the Stream Graphics Processor 14 are stored. Slopesof monotone increases Vh1 and Vh2 are based on transfer rate Rd from theStream Graphics Processor 14 to the Object Buffer 15. A decode period ofeach of ODS1 and ODS2 corresponds to a time period in which a monotonedecrease occurs on the third level and a monotone increase occurs on thesecond level. The beginning of the decode period is shown by a DTS ofthe ODS, whereas the end of the decode period is shown by a PTS of theODS. Once the uncompressed graphics Object has been transferred to theObject Buffer 15 by the time shown by the PTS of the ODS, the decodingof the ODS is complete. It is essential for the uncompressed graphicsObject to be stored in the Object Buffer 15 by the time shown by the PTSof the ODS. As long as this is satisfied, the monotone decrease and themonotone increase in the decode period are not limited to those shown inFIG. 32.

The first level shows changes in occupancy of the Graphics Plane 8. Asillustrated, the changes in occupancy of the Graphics Plane 8 includemonotone increase Vf3 with which the graphics Objects output from theObject Buffer 15 are stored. A slope of monotone increase Vf3 is basedon transfer rate Rc from the Object Buffer 15 to the Graphics Plane 8.The end of monotone increase Vf3 is shown by the PTS of the PCS.

The graph such as the one shown in FIG. 32 can be created through theuse of the DTSs and PTSs of the ODSs, the DTS and the PTS of the PCS,and the buffer sizes and transfer rates shown in FIG. 29. Such a graphallows the author to grasp how the buffer states change when the AV Clipon the BD-ROM 100 is reproduced.

These changes of the buffer states can be adjusted by rewriting DTSs andPTSs. Therefore, it is possible for the author to prevent the occurrenceof such a decoding load that exceeds specifications of a decoder of thereproduction apparatus 200, or to prevent a buffer overflow duringreproduction. This makes it easier to implement hardware and softwarewhen developing the reproduction apparatus 200. This completes theexplanation on the internal construction of the reproduction apparatus200.

The following explains how to implement the Graphics Decoder 12. TheGraphics Decoder 12 can be realized by having a general-purpose CPUexecute a program for performing an operation shown in FIG. 33. Anoperation of the Graphics Decoder 12 is explained below, with referenceto FIG. 33.

FIG. 33 is a flowchart showing an operation of loading a functionalSegment. In the drawing, SegmentK is a variable indicating a Segment(PCS, WDS, PDS, or ODS) which belongs to a DS and is read duringreproduction of the AV Clip, and an ignore flag indicates whetherSegmentK is to be ignored or loaded. In this flowchart, after the ignoreflag is reset to 0 (S20), a loop of steps S21 to S24 and S27 to S31 isperformed for each SegmentK (S25 and S26).

Step S21 judges whether SegmentK is a PCS. If SegmentK is a PCS, theoperation proceeds to step S27.

Step S22 judges whether the ignore flag is 0 or 1. If the ignore flag is0, the operation proceeds to step S23. If the ignore flag is 1, theoperation proceeds to step S24. In step S23, SegmentK is loaded to theCoded Data Buffer 13.

If the ignore flag is 1 (S22:NO), SegmentK is ignored (S24). This leadsto the negative judgement on all functional Segments belonging to the DSin step S22, as a result of which the functional Segments of the DS areall ignored.

Thus, the ignore flag indicates whether SegmentK is to be ignored orloaded. Steps S27 to S31 and S34 to S35 are performed to set this ignoreflag.

Step S27 judges whether a composition_state field of the PCS showsAcquisition Point. If the composition_state field shows AcquisitionPoint, the operation proceeds to step S28. If the composition_statefield shows Epoch Start or Normal Case, the operation proceeds to stepS31.

Step S28 judges whether an immediately preceding DS exists in any of thebuffers (the Coded Data Buffer 13, the Stream Graphics Processor 14, theObject Buffer 15, and the Composition Buffer 16) in the Graphics Decoder12. The immediately preceding DS does not exist in the Graphics Decoder12 if a skip operation is performed. In this case, the display needs tobe started from the Acquisition Point DS, so that the operation proceedsto step S30 (S28:NO).

In step S30, the ignore flag is set to 0, and the operation proceeds tostep S22.

On the other hand, the immediately preceding DS exists in the GraphicsDecoder 12 if normal reproduction is performed. In this case, theoperation proceeds to step S29 (S28:YES). In step S29, the ignore flagis set to 1, and the operation proceeds to step S22.

Step S31 judges whether the composition_state field shows Normal Case.If the composition_state field shows Normal Case, the operation proceedsto step S34. If the composition_state field shows Epoch Start, theoperation proceeds to step S30 where the ignore flag is set to 0. StepS34 is the same as step S28, and judges whether the immediatelypreceding DS exists in the Graphics Decoder 12. If the immediatelypreceding DS exists, the ignore flag is set to 0 (S30). Otherwise, theignore flag is set to 1, because enough functional Segments forcomposing one screen of graphics cannot be obtained (S35). In this way,when the immediately preceding DS does not exist in the Graphics Decoder12, the functional Segments of the Normal Case DS are ignored.

The following gives a specific example of loading DSs, with reference toFIG. 34. In FIG. 34, three DSs (DS1, DS10, and DS20) are multiplexedwith video. A composition_state field of DS1 shows Epoch Start, acomposition_state field of DS10 shows Acquisition Point, and acomposition_state field of DS20 shows Normal Case.

Suppose a skip operation is performed on picture data pt10 in an AV Clipin which these three DSs are multiplexed with video, as indicated byarrow am1. In such a case, DS10 which is closest to pt10 is subjected tothe operation shown in FIG. 33. The composition_state field of DS10shows Acquisition Point (S27:YES), but the immediately preceding DS(DS1) does not exist in the Coded Data Buffer 13 (S28:NO). Accordingly,the ignore flag is set to 0 (S30). As a result, DS10 is loaded to theCoded Data Buffer 13 as indicated by arrow md1 in FIG. 35. Suppose, onthe other hand, a skip operation is performed on picture data which islocated after DS10, as indicated by arrow am2 in FIG. 34. In this case,DS20 is a Normal Case DS, and the immediately preceding DS (DS10) doesnot exist in the Coded Data Buffer 13. Accordingly, DS20 is ignored, asindicated by arrow md2 in FIG. 35.

FIG. 37 shows how DS1, DS10, and DS20 are loaded when normalreproduction is performed as shown in FIG. 36. Of the three DSs, DS1which is an Epoch Start DS is loaded to the Coded Data Buffer 13, asindicated by arrowrd1 (S23). However, the ignore flag is set to 1 forDS10 which is an Acquisition Point DS (S29), so that the functionalSegments of DS10 are not loaded to the Coded Data Buffer 13 but ignored,as indicated by arrow rd2 (S24). Meanwhile, DS20 which is a Normal CaseDS is loaded to the Coded Data Buffer 13, as indicated by arrow rd3(S23).

The following explains an operation of the Graphics Controller 17. FIGS.38 to 40 are flowcharts showing the operation of the Graphics Controller17.

Steps S41 to S44 constitute a main routine, where an event specified byany of steps S41 to S44 is waited.

In FIG. 38, step S41 judges whether the current reproduction time is aDTS of a PCS. If so, steps S45 to S53 are performed.

Step S45 judges whether a composition_state field of the PCS shows EpochStart. If so, the entire Graphics Plane 8 is cleared in step S46.Otherwise, a Window specified by a window_horizontal_position field, awindow_vertical_position field, a window_width field, and awindow_height field of a WDS is cleared in step S47.

Step S48 is performed after step S46 or S47, and judges whether a PTS ofarbitrary ODSx has passed. Clearing the entire Graphics Plane 8 takes along time, so that decoding of ODSx may already be completed by the timethe entire Graphics Plane 8 is cleared. Step S48 examines thispossibility. If the PTS of ODSx has not passed, the operation returns tothe main routine. If the PTS of ODSx has passed, steps S49 to S51 areperformed. Step S49 judges whether an object_cropped_flag field shows 0.If so, a graphics Object corresponding to ODSx is set to non-display(S50).

If the object_cropped_flag shows 1, the graphics Object cropped based onan object_cropping_horizontal_position field, anobject_cropping_vertical_position field, a cropping_width field, and acropping_height field is written to the Window on the Graphics Plane 8at a position specified by an object_horizontal_position field and anobject_vertical_position field (S51). In this way, the graphics Objectis written to the Window.

Step S52 judges whether a PTS of another ODS (ODSy) has passed. Ifdecoding of ODSy is completed during when the graphics Object of ODSx isbeing written to the Graphics Plane 8, ODSy is set as ODSx (S53), andthe operation returns to step S49. As a result, steps S49 to S51 areperformed on ODSy.

In FIG. 39, step S42 judges whether the current reproduction time is aPTS of the WDS. If so, the operation proceeds to step S54, to judgewhether the number of Windows is 1. If the number of Windows is 2, theoperation returns to the main routine. If the number of Windows is 1, aloop of steps S55 to S59 is performed. In this loop, steps S57 to S59are performed for each of at most two graphics Objects to be displayedin the Window. Step S57 judges whether the object_cropped_flag fieldshows 0. If so, the graphics Object is set to non-display (S58).

If the object_cropped_flag field shows 1, the graphics Object croppedbased on an object_cropping_horizontal_position field, anobject_cropping_vertical_position field, a cropping_width field, and acropping_height field is written to the Window on the Graphics Plane 8at a position specified by an object_horizontal_position field and anobject_vertical_position field (S59). As a result of this loop, one ormore graphics Objects are written to the Window.

Step S44 judges whether the current reproduction time is a PTS of thePCS. If so, the operation proceeds to step S60 to judge whether apalette_update_flag field shows 1. If so, a Palette identified by apalette_id field is set to the CLUT unit 9 (S61). If thepalette_update_flag field shows 0, step S61 is skipped.

After this, the CLUT unit 9 performs color conversion of graphics on theGraphics Plane 8. The graphics is then overlaid on video (S62).

In FIG. 40, step S43 judges whether the current reproduction time is aPTS of an ODS. If so, the operation proceeds to step S63 to judgewhether the number of Windows is 2. If the number of Windows is 1, theoperation returns to the main routine.

Here, the judgements made in steps S43 and S63 have the followingmeaning. If the number of Windows is 2, two graphics Objects aredisplayed respectively in the two Windows. In such a case, each timedecoding of one ODS is completed, a graphics Object obtained by thedecoding needs to be written to the Graphics Plane 8 (see FIG. 19B).Therefore, if the current reproduction time is the PTS of the ODS andthe number of Windows is 2, steps S64 to S66 are performed to write eachindividual graphics Object to the Graphics Plane 8. Step S64 judgeswhether the object_cropped_flag field shows 0. If so, the graphicsObject is set to non-display (S65).

If the object_cropped_flag field shows 1, the graphics Object croppedbased on an object_cropping_horizontal_position field, anobject_cropping_vertical_position field, a cropping_width field, and acropping_height field is written to a Window on the Graphics Plane 8 ata position specified by an object_horizontal_position field and anobject_vertical_position field (S66). By repeating this process, twographics Objects are written respectively to the two Windows.

Second Embodiment

The second embodiment of the present invention relates to amanufacturing process of the BD-ROM 100 explained in the firstembodiment. FIG. 41 is a flowchart showing the manufacturing process ofthe BD-ROM 100.

The manufacturing process includes a material production step ofrecording video, sound, and the like (S201), an authoring step ofcreating an application format using an authoring device (S202), and apressing step of creating an original master of the BD-ROM 100 andperforming stamping and bonding to complete the BD-ROM 100 (S203).

In this manufacturing process, the authoring step includes steps S204 toS213.

In step S204, control information, Window definition information,Palette definition information, and graphics are generated. In stepS205, the control information, the Window definition information, thePalette definition information, and the graphics are converted tofunctional Segments. In step S206, a PTS of each PCS is set based on atime of a picture to be synchronized with. In step S207, a DTS[ODS] anda PTS[ODS] are set based on the PTS[PCS]. In step S208, a DTS[PCS], aPTS[PDS], a DTS[WDS], and a PTS[WDS] are set based on the DTS[ODS]. Instep S209, changes in occupancy of each buffer in the player model aregraphed. In step S210, a judgement is made as to whether the graphedchanges satisfy constraints of the player model. If the judgement is inthe negative, the DTS and PTS of each functional Segment are rewrittenin step S211. If the judgement is in the affirmative, a graphics streamis generated in step S212, and the graphics stream is multiplexed with avideo stream and an audio stream to form an AV Clip in step S213. Afterthis, the AV Clip is adapted to the Blue-ray Disc Read-Only Format, tocomplete the application format.

Modifications

Though the present invention has been described by way of the aboveembodiments, the present invention is not limited to such. The presentinvention can be realized with any of modifications (A) to (P) below.The invention of each of the claims of this application includesextension and generalization of the above embodiments and theirmodifications below. The degree of extension and generalization dependsupon the state of the art in the technical field of the presentinvention at the time when the present invention was made.

(A) The above embodiments describe the case where the BD-ROM is used asthe recording medium. Main features of the present invention, however,lie in a graphics stream recorded on the recording medium, which doesnot rely on physical characteristics of BD-ROMs. Therefore, the presentinvention is applicable to any recording medium that is capable ofrecording a graphics stream. Examples of such a recording mediuminclude: an optical disc such as a DVD-ROM, a DVD-RAM, a DVD-RW, aDVD-R, a DVD+RW, a DVD+R, a CD-R, or a CD-RW; a magneto-optical disksuch as a PD or an MO; a semiconductor memory card such as aCompactFlash card, a SmartMedia card, a Memory Stick card, aMultiMediaCard, or a PCMCIA card; a magnetic disk such as a flexibledisk, SuperDisk, Zip, or Clik!; a removable hard disk drive such as ORB,Jaz, SparQ, SyJet, EZFley, or Microdrive, and a nonremovable hard diskdrive.

(B) The above embodiments describe the case where the reproductionapparatus decodes an AV Clip on the BD-ROM and outputs the decoded AVClip to the television. As an alternative, the reproduction apparatusmay be equipped with only a BD drive, with the remaining constructionelements being provided in the television. In this case, thereproduction apparatus and the television can be incorporated in a homenetwork connected with an IEEE 1394 connector. The above embodimentsdescribe the case where the reproduction apparatus is connected to thetelevision, but the reproduction apparatus may instead be integratedwith a display device. Also, the reproduction apparatus may include onlythe system LSI (integrated circuit) which constitutes an essential partof processing. The reproduction apparatus and the integrated circuit areboth an invention described in this specification. Accordingly,regardless of whether the reproduction apparatus or the integratedcircuit is concerned, an act of manufacturing a reproduction apparatusbased on the internal construction of the reproduction apparatusdescribed in the first embodiment is an act of working of the presentinvention. Also, any act of assigning with charge (i.e. for sale) orwithout charge (i.e. as a gift), leasing, and importing the reproductionapparatus is an act of working of the present invention. Likewise, anact of offering for assignment or lease of the reproduction apparatususing storefront displays, catalogs, or brochures is an act of workingof the present invention.

(C) Information processing using the programs shown in the flowcharts isactually realized using hardware resources. Accordingly, the programswhich describe the operational procedures shown in the flowcharts arethemselves an invention. The above embodiments describe the case wherethe programs are incorporated in the reproduction apparatus, but theprograms can be used independently of the reproduction apparatus. Actsof working of the programs include (1) an act of manufacturing, (2) anact of assigning with or without charge, (3) an act of leasing, (4) anact of importing, (5) an act of providing to the public via abi-directional electronic communications network, and (6) an act ofoffering for assignment or lease using storefront displays, catalogs, orbrochures.

(D) The time elements of the steps which are executed in a time seriesin each of the flowcharts can be regarded as the necessary elements ofthe present invention. This being so, a reproduction method shown bythese flowcharts is an invention. If the processing shown in eachflowchart is carried out by performing the steps in a time series so asto achieve the intended aim and the intended effect, this is an act ofworking of the recording method of the present invention.

(E) When recording an AV Clip on the BD-ROM, an extension header may beadded to each TS packet in the AV Clip. The extension header is called aTP_extra_header, includes an arrival_time_stamp and acopy_permission_indicator, and has a data length of 4 bytes. TS packetswith TP_extra_headers (hereafter “EX TS packets”) are grouped in unitsof 32 packets, and each group is written to three sectors. One groupmade up of 32 EX TS packets has 6,144 bytes (=32×192), which isequivalent to a size of three sectors that is 6144 bytes (=2048×3). The32 EX TS packets contained in the three sectors are called an AlignedUnit.

In a home network connected with an IEEE 1394 connector, thereproduction apparatus transmits an Aligned Unit in the followingmanner. The reproduction apparatus removes a TP_extra_header from eachof the 32 EX TS packets in the Aligned Unit, encrypts the body of eachTS packet according to the DTCP Specification, and outputs the encryptedTS packets. When outputting the TS packets, the reproduction apparatusinserts an isochronous packet between adjacent TS packets. A positionwhere the isochronous packet is inserted is based on a time shown by anarrival_time_stamp of the TP_extra_header. The reproduction apparatusoutputs a DTCP_descriptor, as well as the TS packets. TheDTCP_descriptor corresponds to a copy_permission_indicator in theTP_extra_header. With the provision of the DTCP_descriptor indicating“copy prohibited”, it is possible to prevent, when using the TS packetsin the home network connected with the IEEE 1394 connector, the TSpackets from being recorded to other devices.

(F) The above embodiments describe the case where an AV Clip of theBlu-ray Disc Read-Only Format is used as a digital stream, but thepresent invention can also be realized with a VOB (Video Object) of theDVD-Video Format or the DVD-Video Recording Format. The VOB is a programstream that complies with the ISO/IEC 13818-1 Standard and is obtainedby multiplexing a video stream and an audio stream. Also, the videostream in the AV Clip may be an MPEG4 video stream or a WMV videostream. Further, the audio stream in the AV Clip may be a Linear PCMaudio stream, a Dolby AC-3 audio stream, an MP3 audio stream, anMPEG-AAC audio stream, or a dts audio stream.

(G) The film described in the above embodiments may be obtained byencoding an analog image signal broadcast by analog broadcasting. Also,the film may be stream data made up of a transport stream broadcast bydigital broadcasting.

Alternatively, an analog/digital image signal recorded on a videotapemay be encoded to obtain content. Also, an analog/digital image signaldirectly captured by a video camera may be encoded to obtain content. Adigital work distributed by a distribution server is applicable too.

(H) Graphics Objects described in the above embodiments is run-lengthencoded raster data. Run-length encoding is used forcompression/encoding of graphics Objects, because the run-lengthencoding is suitable for compression and decompression of subtitles.Subtitles have a property in that a continuous length of the same pixelvalue in a horizontal direction is relatively long. Therefore, byperforming compression using run-length encoding, a high compressionrate can be attained. In addition, run-length encoding reduces a loadfor decompression, and is therefore suitable for realizing decoding bysoftware. Nevertheless, the use of run-length encoding for graphicsObjects is not a limitation of the present invention. For example,graphics Objects may be PNG data. Also, graphics Objects may be vectordata instead of raster data. Further, graphics Objects may betransparent patterns.

(I) Graphics of subtitles selected according to a language setting inthe reproduction apparatus may be subjected to display effects of PCSs.As a result, display effects achieved by using characters which arecontained within the body of video in a conventional DVD can be realizedwith subtitle graphics displayed according to the language setting ofthe reproduction apparatus. This contributes to high practicality.

Also, subtitle graphics selected according to a display setting of thereproduction apparatus may be subjected to display effects of PCSs. Forexample, graphics of various display modes such as wide screen, pan andscan, and letterbox are recorded on the BD-ROM, and the reproductionapparatus selects one of these display modes according to a displaysetting of the television connected with the reproduction apparatus anddisplays corresponding graphics. Since the display effects of PCSs areapplied to such graphics, viewability increases. As a result, displayeffects achieved by using characters which are contained within the bodyof video in a conventional DVD can be realized with subtitle graphicsdisplayed according to the display setting. This contributes to highpracticality.

(J) The first embodiment describes the case where transfer rate Rc fromthe Object Buffer to the Graphics Plane is set so as to clear theGraphics Plane and render graphics on a Window, which is 25% in size ofthe Graphics Plane, within one video frame. However, transfer rate Rcmay be set so that the clearing and the rendering complete within avertical blanking time. Suppose the vertical blanking time is 25% of1/29.97 seconds. Then Rc is 1 Gbps. By setting Rc in this way, graphicscan be displayed smoothly.

Also, writing in sync with line scan can be used together with writingwithin a vertical blanking time. This enables subtitles to be displayedsmoothly with Rc=256 Mbps.

(K) The above embodiments describe the case where the reproductionapparatus includes the Graphics Plane. Alternatively, the reproductionapparatus may include a line buffer for storing uncompressed pixels ofone line. Since conversion to an image signal is performed for eachhorizontal row (line), conversion to an image signal can equally beperformed with the line buffer.

(L) The above embodiments describe the case where graphics is characterstrings representing dialogs in a film, but the present invention isequally applicable to any graphics that is displayed in precisesynchronization with video. Examples of such graphics include anillustration, a pattern, a cartoon character, and a symbol mark.Examples of such graphics also include a combination of figures,letters, and colors constituting a trademark, a national crest, anational flag, a national emblem, a public symbol or seal employed by anational government for supervision/certification, a crest, flag, oremblem of an international organization, or a mark of origin of aparticular item.

(M) The first embodiment describes the case where a Window is providedon the top or bottom of the Graphics Plane, on an assumption thatsubtitles are displayed horizontally on the top or bottom of the screen.Instead, a Window may be provided on the left or right of the GraphicsPlane, to display subtitles vertically on the left or right of thescreen. This enables Japanese subtitles to be displayed vertically.

(O) The Graphics Decoder performs pipeline processing on DSn and DSn+1,when DSn and DSn+1 belong to the same Epoch in the graphics stream. WhenDSn and DSn+1 belong to different Epochs, on the other hand, theGraphics Decoder starts processing DSn+1 after display of graphics ofDSn begins.

Also, there are two types of graphics streams, i.e., a Presentationgraphics stream which is mainly intended to synchronize with video, andan Interactive graphics stream which is mainly intended to realize aninteractive display. The Graphics Decoder performs pipeline processingof DSn and DSn+1 when the graphics stream is a Presentation graphicsstream, and does not perform pipeline processing when the graphicsstream is an Interactive graphics stream.

(P) In the example shown in FIG. 20, graphics data referenced by a PCSin a DS is set before graphics data not referenced by the PCS in the DS.However, a plurality of sets of referenced graphics data and a pluralityof sets of non-referenced graphics data may instead be arranged in a DSin an order specified by their object_ids. Alternatively, the pluralityof sets of referenced graphics data may be arranged in a scan lineorder. The scan line order referred to here is an order in whichscanning is performed. In general, scanning is performed from the topleft to bottom right of a screen. This being so, graphics data whosedisplay coordinates lie at the top left of the screen is set beforegraphics data whose display coordinates lie at the bottom right of thescreen. This accelerates presentation of graphics data.

The present invention can be modified as explained above. Nevertheless,the invention of each of the claims of this application reflects meansfor solving the technical problem encountered by the conventionaltechniques, so that the technical scope of the invention according tothe claims will not extend beyond the technical scope in which oneskilled in the art acknowledges the technical problem. Hence theinvention according to the claims substantially corresponds to thedescription of the specification.

INDUSTRIAL APPLICABILITY

The above embodiments disclose the internal constructions of therecording medium and the reproduction apparatus to which the presentinvention relates, and the recording medium and the reproductionapparatus can be manufactured in volume based on the disclosed internalconstructions. In other words, the recording medium and the reproductionapparatus are capable of being industrially manufactured. Hence therecording medium and the reproduction apparatus have industrialapplicability.

1. A reproduction apparatus for reproducing a digital stream generatedby multiplexing a video stream and a graphics stream, comprising: avideo decoder operable to decode the video stream to generate a movingpicture; a graphics decoder operable to decode the graphics stream togenerate graphics, and a look up table unit, wherein: the graphicsstream is a sequence of packets; the packets include a packet containinggraphics data, a packet containing palette data and a packet containingcontrol data; the graphic data represents a display content using aplurality of code values; the palette data indicates correspondencebetween each of the code values and luminance and color difference thecontrol data has a reference value for a reference to palette data; uponreading control data in the graphics stream, the graphics decoderperforms a control to display, based on the read control data, a graphicthat has been received already; and the graphics decoder performs thedisplay control by setting palette data read from the graphics streaminto the look up table unit, and instructing the look up table unit toperform a color conversion based on the reference value contained in thecontrol data.
 2. A method of recording onto a recording medium,comprising the steps of: generating application data by using anauthoring device; and recording the application data to the recordingmedium, wherein: the application data includes a digital streamgenerated by multiplexing a video stream and a graphics stream; thegraphic stream is a sequence of a plurality of packets; and the packetsinclude a packet containing graphics data, a packet containing palettedata and a packet containing control data; the graphics data representsa display content using a plurality of code values; the palette dataindicates correspondence between each of the code values and luminanceand color difference; the control data indicates that a display color ofa graphics data, which is contained in a preceding packet in thesequence, is to be converted at a predetermined timing, and the controldata has a reference value for a reference to palette data that is to beused in the color conversion.
 3. A computer-readable recording mediumstoring a program for causing a computer to reproduce a digital streamgenerated by multiplexing a video stream and a graphics stream,comprising: program code operable to cause the computer to decode thevideo stream to generate a moving picture; and program code operable tocause the computer to decode the graphics stream to generate graphics,and display the graphics, wherein the graphics stream is a sequence of aplurality of packets; and the packets include a packet containinggraphics data, a packet containing palette data and a packet containingcontrol data; the graphics represents a display content using aplurality of code values; the palette data indicates correspondencebetween each of the code values and luminance and color difference; thecontrol data has a reference value for a reference to palette data; uponreading control data in the graphics stream, the program code forcausing the computer to display the graphics, performs a control todisplay, based on the read control data, the graphics that has beenreceived already; and the program code for causing the computer todisplay the graphics performs the display control by performing a colorconversion using the palette data corresponding to the reference valuecontained in the control data.
 4. A method of causing a computer toreproduce a digital stream generated by multiplexing a video stream anda graphics stream, comprising the steps of: decoding the video stream togenerate a moving picture; and decoding the graphics stream to generategraphics, and displaying the graphics, wherein the graphics stream is asequence of a plurality of packets; and the packets include a packetcontaining graphic data, a packet containing palette data and a packetcontaining control data; the graphics data represents a display contentusing a plurality of code values; the palette data indicatescorrespondence between each of the code values and luminance and colordifference; the control data has a reference value for a reference tothe palette data; upon reading control data in the graphics stream, thegraphics displaying step performs a control to display, based on theread control data, the graphic that has been received already; and thegraphics displaying step performs the display control by performing acolor conversion using the palette data corresponding to the referencevalue contained in the control data.